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Preface 


This  annotated  reading  list  was  developed  over  the  course  of  the  project  “Laying  the 
Foundation  for  the  Assessment  of  Inform,  Influence,  and  Persuade  Efforts,”  which 
sought  to  identify  and  recommend  selected  best  practices  in  assessment  and  evalu¬ 
ation  from  existing  U.S.  Department  of  Defense  (DoD)  practice,  academic  evalua¬ 
tion  research,  public  relations,  public  diplomacy,  and  public  communication  including 
social  marketing.  This  report  is  a  supplement  to  two  companion  volumes,  Assessing  and 
Evaluating  Department  of  Defense  Efforts  to  Inform,  Influence,  and  Persuade:  Desk  Ref¬ 
erence  and  Assessing  and  Evaluating  Department  of  Defense  Efforts  to  Inform,  Influence, 
and  Persuade:  Handbook  for  Practitioners d  It  was  developed  at  the  request  of  a  project 
stakeholder  as  a  topical  reading  list  to  help  increase  the  assessment  and  evaluation 
knowledge  and  expertise  of  relevant  personnel  through  self-study.  It  should  be  of  inter¬ 
est  to  those  within  DoD  who  wish  to  develop  or  expand  their  expertise  on  assessment 
and  evaluation,  especially  with  regard  to  information  operations  or  inform  and  influ¬ 
ence  activities. 

This  research  was  sponsored  by  the  Rapid  Reaction  Technology  Office  in  the 
Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
and  conducted  within  the  International  Security  and  Defense  Policy  Center  of  the 
RAND  National  Defense  Research  Institute,  a  federally  funded  research  and  develop¬ 
ment  center  sponsored  by  the  Office  of  the  Secretary  of  Defense,  the  Joint  Staff,  the 
Unified  Combatant  Commands,  the  Navy,  the  Marine  Corps,  the  defense  agencies,  and 
the  defense  Intelligence  Community,  under  contract  number  W91WAW-12-C-0030. 

For  more  information  on  the  International  Security  and  Defense  Policy  Center, 
see  http://www.rand.org/nsrd/ndri/centers/isdp.html  or  contact  the  director  (contact 
information  is  provided  on  the  web  page). 
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of  Defense  Efforts  to  Inform ,  Influence,  and  Persuade:  Desk  Reference ,  Santa  Monica,  Calif.:  RAND  Corporation, 
RR-809/1-OSD,  2015;  Christopher  Paul,  Jessica  Yeats,  Colin  P.  Clarke,  Miriam  Matthews,  and  Lauren  Skrabala, 
Assessing  and  Evaluating  Department  of  Defense  Efforts  to  Inform,  Influence,  and  Persuade:  Handbook  for  Practitio¬ 
ners ,  Santa  Monica,  Calif.:  RAND  Corporation,  RR-809/2-OSD,  2015. 
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Introduction 


The  U.S.  Department  of  Defense  (DoD)  lacks  personnel  with  sufficient  expertise  in 
assessment  and  evaluation  to  meet  growing  demand.  At  the  time  of  this  writing  in  late 
2014,  numerous  efforts  were  afoot  to  infuse  sound  assessment  principles  in  doctrine 
and  to  expand  assessment-related  course  offerings  in  the  military- academic  sector,  but 
these  efforts  will  take  time  to  bear  fruit. 

This  temporary  gap  extends  to  the  evaluation  and  assessment  of  DoD  efforts  to 
inform,  influence,  and  persuade.  Congressionally  mandated  reporting  requirements  in 
this  area  have  joined  other  imperatives  for  improved  assessment,  but  a  lack  of  qualified 
and  experienced  evaluation  and  assessment  subject-matter  experts  (SMEs)  continues 
to  impede  progress.  To  help  fill  this  gap,  the  assessment  team  at  the  Joint  Informa¬ 
tion  Operations  Warfare  Center  (JIOWC)  provides  assessment  support  to  the  combat¬ 
ant  commands  and  other  stakeholders.  As  part  of  a  broader  research  effort  aimed  at 
improving  assessment  of  efforts  to  inform,  influence,  and  persuade,  the  JIOWC  assess¬ 
ment  team  asked  RAND  to  provide  a  reading  list  for  self-study  in  this  area.  This  read¬ 
ing  list  is  meant  (in  the  short  term)  to  serve  two  purposes:  first,  to  provide  resources  for 
new  personnel  on  the  JIOWC  assessment  team  to  cement  and  broaden  their  assessment 
and  evaluation  expertise,  ensuring  that  they  are  ready  to  provide  needed  SME  input 
to  relevant  stakeholders,  and,  second,  to  provide  a  general  list  of  assessment  resources 
with  relevance  to  efforts  to  inform,  influence,  and  persuade  that  can  be  made  available 
to  assessment  stakeholders  to  support  self-study  and  improve  assessment  expertise. 
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Evaluation,  Assessment,  and  Measurement,  General  Principles 

Resources  in  this  section  are  foundational  to  assessment  and  evaluation,  drawing  on 
the  best  principles  from  academic  evaluation  research.  Even  those  with  years  of  practi¬ 
cal  experience  conducting  assessment  should  find  these  materials  useful. 

Peter  H.  Rossi,  Mark  W.  Lipsey,  and  Howard  E.  Freeman,  Evaluation:  A  Systematic 
Approach ,  7th  ed.,  Thousand  Oaks,  Calif.:  Sage  Publications,  2004. 

This  is  the  single  best  graduate  textbook  on  evaluation  and  measurement.  If  you  had 
to  choose  only  one  item  on  this  list  to  read,  this  should  be  it.  This  book  is  accessible 
and  comprehensible,  yet  comprehensive.  It  is  the  original  source  for  “the  hierarchy  of 
evaluation,”  the  nested  approach  to  evaluation  described  in  the  volumes  that  accom¬ 
pany  this  annotated  reading  list.  Of  the  12  chapters,  the  first  ten  are  particularly  strong 
and  useful. 

Alan  Clarke,  Evaluation  Research:  An  Introduction  to  Principles,  Methods  and  Practice, 
London:  Sage  Publications,  2005. 

This  relatively  short  introductory  graduate-level  text  on  evaluation  research  discusses 
the  different  types  and  purposes  of  evaluation  and  demands  that  evaluation  have  a 
purpose — and  that  purpose  is  to  support  decisionmaking.  Chapter  1  (“Understanding 
Evaluation”),  chapter  2  (“Quantity  and  Quality — Paradigmatic  Choices  in  Evaluation 
Methodology”),  and  chapter  3  (“Methods  of  Data  Collection”)  are  particularly  valu¬ 
able;  later  chapters  offer  case-study  examples  from  further  afield. 

Harry  P.  Hatry,  Performance  Measurement:  Getting  Results,  2nd  ed.,  Washington,  D.C.: 
Urban  Institute  Press,  2007. 

Hatry  is  considered  the  “pioneer”  of  methods  to  measure  effectiveness  in  the  public 
sector.  The  text  provides  a  great  overview  of  all  steps  in  the  evaluation  process — from 
identifying  the  program’s  mission,  objectives,  customers,  and  trackable  outcomes  to 
finding  the  best  indicators  for  each  outcome,  sources  of  data,  and  methods  to  col- 
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lect  them.  Note  that  “performance”  in  the  title  is  not  limited  to  “measures  of  perfor¬ 
mance”  versus  “measures  of  effectiveness”  (and  this  is  not  a  distinction  the  text  is  overly 
concerned  with);  rather,  it  covers  measurement  at  all  levels  at  which  measurement  is 
important. 

UK  Ministry  of  Defence,  Assessment ,  Joint  Doctrine  Note  2/12,  Shriveham,  UK, 
February  2012,  http://www.mod.uk/NR/rdonlyres/30D3468C-75B3-4FD9-944D- 
975D40E3DCAA/0/201 20221jdn212_assessment.pdf. 

This  doctrinal  document  is  an  excellent  application  of  evaluation  research  principles 
to  the  defense  context  that  goes  well  beyond  traditional  defense-related  approaches. 
It  begins  by  extolling  the  importance  of  assessment,  describes  challenges  to  assess¬ 
ment  (such  as  complexity,  multiple  stakeholders  and  interests,  conflict  between  subjec¬ 
tive  and  objective  measures,  and  political  imperatives),  addresses  the  fundamentals  of 
assessment,  and  offers  step-by-step  guidance  for  assessment  design. 

Amelia  Arsenault,  Sheldon  Flimelfarb,  and  Susan  Abbott,  Evaluating  Media 
Interventions  in  Conflict  Countries:  Toward  Developing  Common  Principles  and  a 
Community  of  Practice,  Washington,  D.C.:  United  States  Institute  of  Peace,  2011, 
http://www.usip.org/files/resources/PW77.pdf. 

This  report  is  a  summary  of  the  proceedings  of  the  2010  stakeholder  conference  “Eval¬ 
uating  Media’s  Impact  in  Conflict  Countries,”  held  at  the  Caux  Conference  Center  in 
Switzerland.  It  describes  the  “Caux  Principles”  developed  by  the  stakeholder  partici¬ 
pants  for  improving  the  evaluation  process.  The  principles  are  as  follows: 

1.  Expand  financial  support  for  evaluation  of  media  interventions  in  conflict 
countries. 

2.  Encourage  flexible  program  and  resource  designs  that  are  sensitive  to  changing 
conflict  conditions. 

3.  Carefully  select  conflict-specific  media  indicators. 

4.  Engage  and  collaborate  with  local  researchers  familiar  with  conflict  conditions. 

5.  Foster  learning,  sharing,  and  collaboration  about  evaluation. 

6.  Embed  evaluation  into  the  entire  project’s  life  cycle  and  beyond. 

7.  Promote  realistic  evaluation  practices. 

8.  Work  to  promote  greater  clarity  surrounding  evaluation. 

Issues  related  to  a  lack  of  transparency,  sharing,  and  incentives  for  integrity  and 
honesty  are  key  themes  throughout  the  report,  including  concerns  about  “survey 
fatigue.” 
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Maureen  Taylor,  “Methods  of  Evaluating  Media  Interventions  in  Conflict 
Countries,”  paper  presented  at  “Evaluating  Media’s  Impact  in  Conflict  Countries” 
conferences,  Caux,  Switzerland,  December  13-17,  2010, 
http://www.global.asc.upenn.edu/fileLibrary/PDFs/taylorcaux2.pdf. 

Prepared  for  the  2010  Caux  conference  on  monitoring  and  evaluating  media  interven¬ 
tions,  this  short  paper  identifies  and  examines  the  five  social  science  research  method¬ 
ologies  that  can  be  used  in  monitoring  and  evaluating  higher-level  media  assistance 
programs  in  conflict  environments:  (1)  content  analysis,  (2)  Delphi  panels,  (3)  focus 
groups,  (4)  network  analysis,  and  (5)  survey  research. 

The  Better  Evaluation  Planning  Tool,  http://betterevaluation.org. 

This  evaluation  resource  portal  organizes  more  than  200  evaluation  options  in  a  seven- 
cluster  framework  called  the  Rainbow  Framework  (manage,  define,  frame,  describe, 
understand  causes,  synthesize,  and  report  and  support  use).  There  are  many  useful 
materials  here,  starting  with  the  framework  itself. 

Michael  Quinn  Patton,  Qualitative  Research  and  Evaluation  Methods,  3rd  ed., 
Thousand  Oaks,  Calif.:  Sage  Publications,  2002. 

This  book  provides  a  thorough  review  of  the  different  types  of  qualitative  data,  the 
strengths  and  weaknesses  of  qualitative  data,  and  research  methods  for  collecting  these 
data.  In  addition,  it  briefly  describes  a  range  of  approaches  to  the  analysis  of  qualitative 
data.  Mastering  this  information  would  support  the  collection  and  use  of  qualitative 
data  for  assessment  purposes. 


Identifying  and  Developing  Objectives 

One  key  assessment  principle  that  emerges  in  all  strong  thinking  on  the  subject  is  the 
importance  of  clear  objectives.  While  many  of  the  items  in  the  previous  section  make 
this  point  and  offer  related  advice,  this  section  features  a  single  item  almost  wholly 
concerned  with  this  issue. 

Ralph  L.  Keeney,  “Developing  Objectives  and  Attributes,”  in  Ward  Edwards, 

Ralph  F.  Miles,  Jr.,  and  Detlof  von  Winterfeldt,  eds.,  Advances  in  Decision  Analysis: 
From  Foundations  to  Applications,  New  York:  Cambridge  University  Press,  2007, 
pp.  104-128. 

This  sometimes  overly  academic  chapter  recognizes  that  specifying  objectives  is  not 
always  as  easy  as  it  seems  it  should  be  and  offers  advice  for  getting  from  values  to  objec¬ 
tives,  and  for  the  qualities  those  objectives  should  have  (complete,  nonredundant,  con¬ 
cise,  specific,  and  understandable).  It  also  describes  a  somewhat  practical  procedure  for 
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working  with  SMEs  and  stakeholders  to  refine  objectives,  working  iteratively  between 
organizational  values  and  specific  objectives.  Further,  the  chapter  includes  a  process  for 
developing  measures  (referred  to  as  “attributes”)  for  objectives. 


Logic  Models  and  Theories  of  Change 

Having  a  theory  of  change — a  set  of  assumptions  or  beliefs  connecting  the  specific 
activities  of  a  program  or  other  effort  to  its  objectives — is  foundational  for  effective 
assessment.  These  works  elaborate  the  role  of  a  theory  of  change  and  describe  a  popular 
way  to  represent  it,  the  logic  model. 

Sue  C.  Funnell  and  Patricia  J.  Rogers,  Purposeful  Program  Theory:  Effective  Use  of 
Theories  of  Change  and  Logic  Models,  San  Francisco,  Calif.:  Jossey-Bass,  2011. 

Regarded  by  evaluation  experts  as  a  valuable  resource  for  program  logic  modeling,  the 
text  states  that  “program  theory  provides  a  coherent  picture  of  how  change  occurs  and 
how  to  improve  performance.  .  .  .  [This  book]  shows  how  to  develop,  represent,  and  use 
program  theory  ...  to  suit  your  particular  situation.” 

W.  K.  Kellogg  Foundation,  Logic  Model  Development  Guide:  Using  Logic  Models  to 
Bring  Together  Planning,  Evaluation,  and  Action,  Battle  Creek,  Mich.,  January  2004. 

This  excellent  and  relatively  short  monograph  introduces  logic  modeling  and  walks  the 
reader  through  the  process  of  developing  a  theory  of  change  logic  model  for  a  program. 


Evaluation  Capacity  Building 

Evaluation  capacity  building  is  an  intentional  process  to  increase  individual  motiva¬ 
tion,  knowledge,  and  skills  to  enhance  a  group  or  organization’s  ability  to  conduct  or 
use  evaluation. 

Abraham  Wandersman,  “Getting  To  Outcomes:  An  Evaluation  Capacity  Building 
Example  of  Rationale,  Science,  and  Practice,”  American  Journal  of  Evakiation, 

Vol.  35,  No.  1,  March  2014,  pp.  100-106. 

This  short  article  offers  a  frame  and  empirical  results  for  operationalizing  the  eval¬ 
uation  capacity  building  model  using  the  results-based  approach  of  Getting  To 
Outcomes,  which  is  designed  to  enhance  practitioner  capacity.  The  Getting 
To  Outcomes  approach  involves  ten  accountability-related  steps,  such  as  assessing 
needs  and  resources,  setting  goals  and  desired  outcomes,  and  selecting  evidence-based 
practices. 
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J.  Bradley  Cousins,  Swee  C.  Goh,  Catherine  J.  Elliott,  and  Isabelle  Bourgeois, 
“Framing  the  Capacity  to  Do  and  Use  Evaluation,”  in  J.  Bradley  Cousins  and  Isabelle 
Bourgeois,  eds.,  Organizational  Capacity  to  Do  and  Use  Evaluation:  New  Directions 
for  Evaluation,  New  Directions  for  Evaluation  series,  No.  141,  San  Francisco,  Calif.: 
Jossey-Bass,  Spring  2014,  pp.  7-23. 

The  authors  review  evolving  conceptions  of  evaluation  capacity  building  and  put  for¬ 
ward  a  guiding  framework  for  organizational  capacity  for  evaluation.  The  framework 
addresses  not  only  the  capacity  to  do  but  also  the  capacity  to  use  evaluation.  It  covers 
such  topics  as  utilization  and  evaluation  theory,  professional  standards  of  practice, 
evaluation  and  organizational  learning,  and  direct  versus  indirect  capacity  building. 


Traditional  Defense  Assessment 

There  is  a  traditional  and  doctrinal  way  to  assess  military  operations.  There  are  many 
good  principles  espoused  in  this  tradition,  and  it  should  not  be  overlooked.  Elowever, 
many  of  the  foundational  insights  of  evaluation  research  have  failed  to  penetrate  assess¬ 
ment  doctrine.  While  this  is  a  place  to  look  for  enrichment  and  comprehensiveness,  it 
is  not  a  source  for  all  the  answers. 

P.  T.  Eles,  E.  Vincent,  B.  Vasiliev,  and  K.  M.  Banko,  Opinion  Polling  in  Support  of 
the  Canadian  Mission  in  Kandahar:  A  Final  Report  for  the  Kandahar  Province  Opinion 
Polling  Program,  Including  Program  Overview,  Lessons,  and  Recommendations,  Ottawa, 
Ont.:  Defence  R&D  Canada,  Centre  for  Operational  Research  and  Analysis,  DRDC 
CORA  TR  2012-160U,  September  2012. 

This  report  provides  an  excellent  summary  of  best  practices  in  survey  research  in  sup¬ 
port  of  counterinsurgency  operations,  drawn  from  the  authors’  experience  managing 
the  Kandahar  Province  Opinion  Polling  (KPOP)  program.  Lessons  learned  and  rec¬ 
ommendations  fall  into  the  following  categories:  polling  program  initiation,  manage¬ 
ment,  and  oversight,  including  the  roles  and  responsibilities  of  the  various  organizations 
involved  in  the  survey  program;  survey  questionnaire  formulation  and  management; 
survey  design  and  sampling  strategy;  data  analysis;  and  the  presentation  and  interpre¬ 
tation  of  results.  A  classified  version  of  the  report  includes  annexes  on  the  results  of  the 
three  KPOP  polls  conducted  in  2011. 

U.S.  Joint  Chiefs  of  Staff,  Commander’s  Handbook  for  Assessment  Planning  and 
Execution,  version  1.0,  Suffolk,  Va.:  Joint  Staff,  J7,  Joint  and  Coalition  Warfighting, 
September  9,  2011. 

This  handbook  offers  a  useful  summary  of  the  traditional  military  view  of  assessment, 
including  much  of  what  is  taught  as  assessment  in  DoD.  While  the  handbook  has 
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many  shortcomings  (i.e.,  it  skips  over  practical  execution  details  and  misses  some  of  the 
critical  insights  from  evaluation  research),  it  provides  a  good  foundation. 


Evaluation  of  Public  Communication 

Public  communication  campaigns  are  efforts  to  inform,  influence,  and  persuade, 
though  not  traditionally  in  the  defense  context.  While  not  perfectly  analogous,  lessons 
for  the  evaluation  and  assessment  of  public  communication  certainly  have  merit  for 
our  purposes. 

Thomas  W.  Valente,  Evaluating  Health  Promotion  Programs,  Oxford,  UK:  Oxford 
University  Press,  2002. 

Valente  provides  an  excellent  review  of  designs  and  methods  used  to  evaluate  the  per¬ 
formance  and  effectiveness  of  public  communication  campaigns.  Drawing  from  years 
of  experience  conducting  evaluations  of  large  health  promotion  programs  in  the  devel¬ 
oping  world,  the  concepts  and  best  practices  described  in  the  text  are  highly  relevant  to 
the  circumstances  that  DoD  practitioners  are  likely  to  encounter  in  assessing  informa¬ 
tion  operations  (IO).  It  is  written  with  the  level  of  detail  appropriate  for  an  evaluator 
with  basic  training  in  social  science  methods.  The  text  includes  chapters  on  theories 
of  influence,  techniques  for  measuring  exposure,  sampling  strategies,  statistical  analy¬ 
sis,  survey  design,  and  issues  unique  to  formative,  process,  and  summative  research. 
It  includes  detailed  examples  of  survey  instruments  and  other  measures  from  several 
evaluations  led  by  Valente. 

Charles  T.  Salmon  and  Lisa  Murray-Johnson,  “Communication  Campaign 
Effectiveness  and  Effects:  Some  Critical  Distinctions,”  in  Ronald  Rice  and  Charles 
Atkins,  eds.,  Public  Communication  Campaigns,  4th  ed.,  Thousand  Oaks,  Calif.: 

Sage  Publications,  2012,  pp.  99-112. 

This  chapter  provides  evaluators  with  guidance  for  assessing  commonly  overlooked 
outcomes  of  campaigns,  whether  manifest  or  latent,  intended  or  unintended.  Cam¬ 
paign  effectiveness  is  a  multifaceted  construct  and  is  increasingly  defined  using  relative 
rather  than  absolute  standards.  The  authors  make  distinctions  among  various  types 
of  campaign  effectiveness,  including  definitional  effectiveness  (e.g.,  getting  a  social 
phenomenon  defined  as  a  social  problem  or  elevating  it  on  the  public  agenda),  con¬ 
textual  effectiveness  (e.g.,  impact  within  particular  contexts  such  as  education  versus 
enforcement  versus  engineering),  cost-effectiveness  comparison  (e.g.,  prevention  versus 
treatment,  addressing  certain  problems  over  others),  and  programmatic  effectiveness 
(e.g.,  testing  campaign  outcomes  relative  to  stated  goals  and  objectives).  Further,  they 
note  that  the  phrases  campaign  effects  and  campaign  effectiveness  have  often  been  used 
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interchangeably  in  evaluation  reports,  obscuring  the  fact  that  some  campaigns  induce 
harmful  or  unintended  effects. 

Thomas  W.  Valente  and  Patchareeya  P.  Kwan,  “Evaluating  Communication 
Campaigns,”  in  Ronald  Rice  and  Charles  Atkins,  eds.,  Public  Communication 
Campaigns ,  4th  ed.,  Thousand  Oaks,  Calif.:  Sage  Publications,  2012,  pp.  83-98. 

This  chapter  includes  sections  on  the  following  topics: 

•  the  nature  of  campaign  evaluation 

•  an  evaluation  framework  that  delineates  the  steps  in  the  evaluation  process  (needs 
assessment,  formative  research,  design,  data  collection,  analysis,  interpretation, 
and  dissemination) 

•  summative  evaluation  research,  covering  theoretical  considerations,  study  designs, 
levels  of  assignment,  sample  sizes,  measures  of  campaign  exposure,  and  measures 
of  impact. 

Valente  and  Kwan  present  six  evaluation  study  designs  to  measure  public  com¬ 
munication  outcomes:  (1)  postcampaign  only;  (2)  pre-  and  postcampaign  compari¬ 
son;  (3)  pre-  and  postcampaign  comparison  and  postcampaign-only  control  group; 
(4)  pre-  and  postcampaign  comparison  and  predetermined  control  group;  (5)  pre-  and 
postcampaign  comparison,  with  predetermined  control  group  and  post-only  interven¬ 
tion  group;  and  (6)  Solomon  four  group.  The  authors  delineate  between  theory  failure 
and  program  failure  and  stress  the  importance  of  linking  a  validated  theory  to  program 
implementation  to  avoid  instances  of  theory  failure. 

Julia  Coffman,  Public  Communication  Campaign  Evaluation:  An  Environmental  Scan 
of  Challenges,  Criticisms,  Practice,  and  Opportunities ,  Washington,  D.C.: 
Communications  Consortium  Media  Center,  2002, 
http://earlygradereading.unitedway.org/sites/default/files/hles/Public%20 
Communication%20Campaign%20Evaluation.pdf. 

This  report  examines  public  communication  evaluation  practices  and  challenges  and 
includes  sections  on  relevant  theory,  outcomes,  and  useful  methods  for  designing  eval¬ 
uations.  Coffman  distinguishes  among  four  types  of  evaluation  in  two  broad  catego¬ 
ries:  “front-end”  evaluations  (formative)  and  “back-end”  evaluations  (process,  outcome, 
and  impact  evaluations).  Outcome  evaluations  measure  “effects”  that  come  about  in 
the  target  population  as  a  result  of  the  campaign  and  can  include  the  following  mea¬ 
sures:  knowledge/awareness,  salience,  attitudes,  norms,  self-efficacy,  behavioral  inten¬ 
tions,  behavior,  skills,  environmental  constraints,  media  change,  and  policy  change. 
Effective  methods  for  outcome  evaluation  include  framing  analysis  and  rolling  sample 
surveys.  Impact  evaluations  capture  the  long-term  outcomes  of  behavior  change  and 
system-level  outcomes.  Impact  evaluation  methods  include  repeated  measures,  staged 
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implementation,  natural  variations  in  treatment,  and  self-determination  of  exposure. 
Chapter  3  (which  addresses  evaluation  challenges  and  criticisms),  chapter  4  (on  cam¬ 
paign  outcomes,  measures,  and  evaluation  methods),  and  chapter  5  (on  opportunities 
for  improving  evaluation  tools)  are  particularly  useful. 


Evaluation  in  Public  Diplomacy 

Public  diplomacy  also  has  a  great  deal  in  common  with  defense  efforts  to  inform,  influ¬ 
ence,  and  persuade,  along  with  applicable  lessons. 

Robert  Banks,  A  Resource  Guide  to  Public  Diplomacy  Evaluation,  CPD  Perspectives 
on  Public  Diplomacy,  Paper  9,  Los  Angeles,  Calif.:  Figueroa  Press,  2011, 
http://uscpublicdiplomacy.org/publications/perspectives/CPD_Perspectives_ 
Paper%209_201 1  .pdf. 

Written  by  the  former  diplomat-in-residence  at  the  Annenberg  School’s  Center  on 
Public  Diplomacy  at  the  University  of  Southern  California,  this  guide  offers  one  of  the 
most  comprehensive  set  of  public  diplomacy  evaluation  resources  available.  Relevant 
sections  include  “The  Administration  of  PD  [Public  Diplomacy]  Evaluation,”  “The 
Evaluation  Process,”  “PD  Metrics  in  the  U.S.  Government,”  “Cultural  Programming,” 
“Information  Campaigns  and  Media  Agenda  Setting,”  “New  Media,”  “PD  and  PR: 
The  Private  Sector  as  a  Model  for  PD  Practitioners,”  and  “PD,  Polling,  and  Audience 
Research.” 

NATO  Joint  Analysis  and  Lessons  Learned  Centre,  A  Framework  for  the  Strategic 
Planning  and  Evaluation  of  Public  Diplomacy,  Lisbon,  Portugal,  2013, 
http://www.jallc.nato.int/newsmedia/docs/A%20Framework%20for%20the%20 
Strategic%20Planning%20and%20Evolution%20of%20Public%20Diplomacy.pdf. 

This  report  provides  a  thorough  description  of  the  concepts  involved  in  public  diplo¬ 
macy  evaluations,  which  are  applicable  to  evaluations  in  other  areas,  especially  the 
assessment  of  efforts  to  inform,  influence,  and  persuade.  It  briefly  discusses  logic 
models,  strategic  planning,  reporting,  and  research  methods.  It  is  a  thorough  guide 
that  may  assist  individuals  in  DoD  in  the  design  and  implementation  of  diverse  efforts. 


Assessment  in  Advertising,  Marketing,  and  Public  Relations 

Frankly,  much  of  the  instructional  material  on  measurement  in  marketing  and  public 
relations  is  only  tangentially  useful  for  the  assessment  of  government  efforts  to  inform, 
influence,  or  persuade  because  of  its  business/bottom-line  focus  and  its  general  opacity. 
There  are  some  good  ideas,  and  some  can  be  translated  and  applied,  but  most  of  the 
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literature  is  not  directly  applicable.  There  is  still  material  worth  reading,  but  the  read¬ 
ings  in  this  section  may  be  given  a  lower  priority. 

Paul  W.  Farris,  Neil  T.  Bendle,  Phillip  E.  Pfeifer,  and  David  J.  Reibstein,  Marketing 
Metrics :  The  Definitive  Guide  to  Measuring  Marketing  Performance,  2nd  ed.,  Upper 
Saddle  River,  N.J.:  Pearson  Education,  2010. 

This  traditional  marketing  measurement  text  argues,  first,  that  measuring  marketing  is 
important  to  a  firm  and  then  discusses  measurement  practice.  This  text  focuses  primar¬ 
ily  on  showing  the  monetary  value  of  marketing  activities,  a  metric  that  makes  sense 
in  industry  but  is  much  less  obviously  relevant  in  government  and  military  contexts. 

Mark  Jeffery,  Data-Driven  Marketing:  The  15  Metrics  Everyone  in  Marketing  Should 
Know,  Hoboken,  N.J.:  John  Wiley  and  Sons,  2010. 

This  volume  contains  some  good  insights  about  data  sources  and  the  importance  of 
assessing  “squishy”  (difficult-to-measure)  things  based  on  data.  The  actual  metrics 
mostly  lack  defense  analogies,  however,  and  can  be  most  appropriately  characterized  as 
highly  business-focused. 

David  Michaelson  and  Don  W.  Stacks,  “Standardization  in  Public  Relations  Mea¬ 
surement  and  Evaluation,”  Public  Relations  Journal,  Vol.  5,  No.  2,  Spring  2011. 

The  authors  argue  that  the  consensus  on  standard  measures  for  public  relations  (PR) 
performance  has  been  elusive  for  two  main  reasons.  First,  there  is  a  general  belief  that 
PR  is  somehow  “different”  and  thus  no  standard  is  applicable,  and,  second,  many  com¬ 
munication  consultants  want  to  own  proprietary  or  “black  box”  measures  that  cannot 
be  independently  replicated  or  tested  for  reliability  and  validity.  The  authors  assert  that 
the  best  approach  to  developing  effective  PR  measurement  includes  the  acceptance  of 
measures  that  are  valid  for  determining  the  impact  of  PR  and  research  methods  and 
that  will  produce  reliable  and  replicable  results.  This  means  combining  what  one  needs 
to  know  (standards)  with  how  to  collect  data  (best  practices). 

David  Michaelson  and  Sandra  Macleod,  “The  Application  of  Best  Practices  in  Public 
Relations  Measurement  and  Evaluation  Systems,”  Public  Relations  Journal,  Vol.  1, 

No.  1,  October  2007. 

This  article  argues  that  adhering  to  the  “best  practices”  in  PR  assessment,  measure¬ 
ment,  and  evaluation  can  be  boiled  down  to  the  following  three  steps: 

•  setting  clear  and  well-defined  research  objectives 

•  applying  a  rigorous  research  design  that  meets  the  highest  standards  of  research 
methods  and  ensures  reliable  research  results 

•  providing  detailed  supporting  documentation  with  full  transparency. 
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Don  W.  Stacks,  Primer  of  Public  Relations  Research,  2nd  ed.,  New  York:  Guilford 
Press,  2011. 

The  author  states  that  how  we  define  what  we  are  measuring  determines  both  how 
precise  we  will  be  in  explaining  what  we  have  observed  and  which  analytical  tools  are 
appropriate  for  the  type  of  measurement  we  need. 

David  Michaelson  and  Toni  Griffin,  A  New  Model  for  Media  Content  Analysis, 
Gainesville,  Fla.:  Institute  for  Public  Relations,  2005. 

PR  professionals  currently  rely  on  nine  basic  methods  of  content  analysis:  clip  count¬ 
ing,  circulation  and  readership  analysis,  advertising  value  equivalence,  simple  content 
analysis,  message  analysis,  tonality  analysis,  prominence  analysis,  quality  of  coverage, 
and  competitive  analysis.  The  authors  argue  that  these  traditional  methods  have  fun¬ 
damental  flaws.  Instead  of  relying  on  these  methods,  a  better  approach  would  be  to 
conduct  content  analysis  to  determine  the  presence  of  four  factors:  correct  information, 
incorrect  information,  misleading  information,  and  omitted  information. 

Russell  H.  Colley,  Defining  Advertising  Goals  for  Measured  Advertising  Results, 

New  York:  Association  of  National  Advertisers,  1961. 

An  interviewed  SME  recommended  applying  the  Defining  Advertising  Goals  for  Mea¬ 
sured  Advertising  Results  framework  as  presented  in  this  classic  work  to  the  assessment 
of  defense  efforts  to  inform,  influence,  and  persuade.  This  application  would  involve 
first  developing  a  model  of  influence  with  specific  goals,  then  tying  measures  to  those 
goals,  measuring,  and  revising.  The  process  is  iterative,  and  one  must  be  ready  to  revise 
goals,  influence  models,  and  measures  based  on  results.  According  to  the  framework, 
information  influences  behavior  through  four  steps:  awareness,  comprehension,  con¬ 
viction,  and  action. 


Thinking  Outside  the  Box:  Truly  Different  Approaches  to 
Measurement  and  Assessment 

In  addition  to  sources  from  the  main  schools  of  thought  on  evaluation  and  assess¬ 
ment,  there  are  a  few  cats  and  dogs  worth  mentioning.  We  offer  neither  of  these  as  “the 
answer”  but  simply  as  additional  ways  of  thinking  about  the  assessment  challenge  that 
might  be  useful  in  some  contexts. 

Douglas  W.  Flubbard,  How  to  Measure  Anything:  Finding  the  Value  of  “Intangibles”  in 
Business,  Hoboken,  N.J.:  John  Wiley  and  Sons,  2010. 

This  interesting  and  insightful  book  presents  a  novel  paradigm  for  assessing  literally 
anything.  It  suggests  a  totally  different  way  of  thinking  about  measurement:  that  mea- 
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surement  is  only  of  value  if  it  improves  your  ability  to  make  a  decision.  Focusing  on 
uncertainty  and  risk,  it  argues  that  the  value  of  assessment  is  to  reduce  the  uncer¬ 
tainty  about  information  on  which  decisions  will  be  made.  It  contains  two  key  insights 
that  might  help  when  the  cost  of  a  proposed  assessment  threatens  to  eclipse  its  value: 
(1)  sometimes,  an  objectively  poor  measurement  still  decreases  uncertainty  and 
improves  decisionmaking,  and  (2)  many  measurements  do  not  add  value  because  the 
cost  of  measuring  them  is  greater  than  the  benefit  of  reducing  uncertainty  for  that 
decision. 

Steve  R.  Corman  and  Kevin  J.  Dooley,  Strategic  Communication  on  a  Rugged- 
Landscape:  Principles  for  Finding  the  Right  Message,  Phoenix,  Ariz.:  Consortium  for 
Strategic  Communication,  Arizona  State  University,  Report  No.  0801,  January  7, 
2008,  http://csc.asu.edu/wp-content/uploads/pdf/121.pdf. 

The  authors  propose  an  evolutionary  approach  to  determining  what  works  in  strategic 
communication — an  iterative  process  that  keeps  what  works  and  adds  random  varia¬ 
tion.  On  the  plus  side,  it  is  aimed  at  strategic  communication,  so  it  fits  cleanly  into 
the  inform,  influence,  and  persuade  interest  area.  On  the  minus  side,  it  is  abstract,  not 
always  completely  accessible,  and  not  entirely  practical.  Interesting,  regardless. 


Evaluation  and  Assessment  Case  Studies  and  Examples 

The  items  in  this  section  are  often  less  explicit  about  the  evaluation  or  assessment  prin¬ 
ciples  being  followed,  but  all  are  practical  examples  of  evaluation  or  assessment.  Fur¬ 
ther,  all  are  examples  from  the  inform,  influence,  and  persuade  domain,  though  mostly 
outside  the  defense  sector. 

Marie-Louise  Mares  and  Zhongdang  Pan,  “Effects  of  Sesame  Street:  A  Meta-Analysis 
of  Children’s  Learning  in  15  Countries,”  Journal  of  Applied  Developmental  Psychology, 
Vol.  34,  No.  3,  May-June  2013,  pp.  140-151. 

Mares  and  Pan  examine  the  effects  of  children’s  exposure  to  international  coproduc¬ 
tions  of  Sesame  Street  through  a  meta-analysis  of  24  studies,  conducted  with  more 
than  10,000  children  in  15  countries,  predominantly  in  the  developing  world.  The 
24  studies  included  four  experimental  designs,  seven  quasi-experimental  designs,  and 
ten  cross-sectional  or  panel  surveys.  Study  results  suggest  that  exposure  to  the  pro¬ 
grams  significantly  enhanced  learning  outcomes  across  three  categories:  cognitive  out¬ 
comes,  learning  about  the  world,  and  social  reasoning  and  attitudes  toward  out-groups. 
The  results  are  “contextualized  by  considering  the  effects  and  reach  of  the  program, 
relative  to  other  early  childhood  interventions.”  The  article  is  particularly  useful  to  IO 
evaluators  because  it  describes  key  elements  of  each  study  design.  In  doing  so,  it  puts 


14  Assessing  and  Evaluating  DoD  Efforts  to  Inform,  Influence,  and  Persuade:  Annotated  Reading  List 


forward  a  “quality  index”  to  rate  the  quality  of  each  study  in  terms  of  its  design  and 
methods — specifically,  the  extent  to  which  the  study  included  the  following: 

•  random  sampling  or  assignment  at  the  individual  level 

•  multiple  indicators  for  key  variables 

•  reliability  assessment  for  key  indexes 

•  quality  control  in  held  operations 

•  experimental  or  statistical  controls 

•  a  strong  basis  for  causal  inferences  (panel  design,  between-group  or  pre/post 
experimental  design). 

Joyee  S.  Chattarjee,  Anurudra  Bhanot,  Lauren  B.  Frank,  Sheila  T.  Murphy,  and 
Gerry  Power,  “The  Importance  of  Interpersonal  Discussion  and  Self-Efficacy  in 
Knowledge,  Attitude,  and  Practice  Models,”  International  Journal  of  Communication, 
Vol.  3,  2009,  pp.  607-634. 

This  article  illustrates  the  use  of  structural  equation  modeling  for  assessing  effects  and 
for  teasing  out  the  relative  contributions  of  various  mediators  of  behavioral  change 
associated  with  communication  campaigns  in  the  developing  world.  Chattarjee  and 
colleagues  used  data  from  a  survey  of  834  sexually  active  men  to  assess  the  influ¬ 
ence  of  exposure  to  a  BBC  program  on  awareness,  attitudes,  and  behaviors  toward 
HIV  and  AIDS.  Respondents  were  matched  based  on  gender,  age,  education,  and 
location.  Using  structural  equation  modeling,  the  researchers  were  able  to  show  that 
people  exposed  to  the  campaign  had  higher  awareness  and  knowledge  of  HIV/AIDS- 
related  issues  and  that  knowledge  change  predicted  attitudinal  change,  which  pre¬ 
dicted  behavioral  change. 

Marie  Gaarder  and  Jeannie  Annan,  Impact  Evaluation  of  Conflict  Prevention  and 
Peacebuilding  Interventions,  Policy  Research  Working  Paper  No.  6469,  New  York: 
World  Bank  Independent  Evaluation  Group,  2013. 

Gaarder  and  Annan  describe  several  quasi-experimental  and  experimental  designs  used 
to  evaluate  World  Bank  conflict  prevention  and  peacebuilding  interventions,  including 
media  interventions.  Evaluations  reviewed  include  the  following: 

•  A  group-based  randomized  experiment  led  by  Elizabeth  Levy  Paluck  evaluated  a 
reconciliation-themed  radio  soap  opera  in  Rwanda,  which  used  matched-pair  ran¬ 
domization  at  the  level  of  listening  groups.  Communities  were  sampled  according 
to  political,  regional,  and  ethnic  criteria  and  then  matched  into  pairs  with  indi¬ 
viduals  from  a  similar  community  on  several  observable  characteristics,  such  as 
gender  ratio,  quality  of  dwelling,  and  education  level.  Then,  “one  community  in 
each  pair  was  randomly  assigned  to  the  reconciliation  program  and  the  other  to 
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the  health  program.  This  stratification  of  sites  helped  to  balance  and  minimize 
observable  differences  between  the  communities  ex  ante.” 

•  A  study  led  by  Paluck  used  randomized  pairwise  matching  within  clusters  to  eval¬ 
uate  the  impact  of  a  radio  soap  opera  when  aired  in  conjunction  with  a  talk  show 
that  emphasized  conflict  reduction  through  community  cooperation.  Paluck  pair¬ 
wise  matched  regions  and  randomly  chose  one  treatment  and  one  control  region 
in  each  pair.  While  the  radio  program  was  aired  in  all  regions  in  the  experiment, 
the  talk  show  following  the  radio  show,  designed  to  encourage  listener  reactions 
and  discussions,  was  broadcast  only  in  treatment  regions.  Paluck  found  that  the 
listeners  exposed  to  the  additional  talk  show  did  discuss  more  but  were  also  more 
likely  to  become  intolerant  and  less  likely  to  help  outcast  community  members. 

•  A  quasi-experimental  evaluation  by  Michael  Gilligan  and  colleagues  exploited  a 
random  disruption  in  program  implementation  to  construct  a  control  group  to 
evaluate  the  impact  of  a  reintegration  program  on  ex-combatants  in  Burundi. 
Three  organizations  were  given  contracts  to  administer  the  program,  but  one 
delayed  providing  services  for  a  year  for  reasons  apparently  unrelated  to  predictors 
of  effectiveness.  To  control  for  potential  systematic  differences  between  individu¬ 
als  in  the  control  group,  participants  in  the  treatment  and  control  groups  were 
matched  on  individual  characteristics,  community  characteristics,  and  propensity 
scores. 

Sandra  D.  Lane,  “Television  Minidramas:  Social  Marketing  and  Evaluation  in 
Egypt,”  Medical  Anthropology  Quarterly ,  Vol.  11,  No.  2,  June  1997,  pp.  164-182. 

The  author  describes  how  to  tailor  influence  campaigns  to  a  foreign  audience  and  how 
best  to  evaluate  these  efforts.  The  issues  examined  here  are  public  health  concerns,  but 
the  evaluation  practices  are  cross-cutting. 

Jorge  A.  Montoya,  Charlotte  K.  Kent,  Harlan  Rotblatt,  Jacque  McCright,  Peter  R. 
Kerndt,  and  Jeffrey  D.  Klausner,  “Social  Marketing  Campaign  Significantly  Associ¬ 
ated  with  Increases  in  Syphilis  Testing  Among  Gay  and  Bisexual  Men  in  San  Fran¬ 
cisco,”  Sexually  Transmitted  Diseases,  Vol.  32,  No.  7,  July  2005,  pp.  395-399. 

This  is  a  straightforward  example  of  an  evaluation  of  a  social  marketing  campaign 
geared  toward  a  specific  audience  (gay  men)  to  convince  them  to  take  a  health  pre¬ 
caution  (get  tested  for  syphilis).  IO  assessment  could  benefit  from  similar  evaluation 
techniques. 
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Joie  Acosta,  Laurie  T.  Martin,  Michael  P.  Fisher,  Racine  Harris,  and  Robin  M. 
Weinick,  Assessment  of  the  Content,  Design,  and  Dissemination  of  the  Real  Warriors 
Campaign ,  Santa  Monica,  Calif.:  RAND  Corporation,  TR-1176-OSD,  2012, 
http://www.rand.org/pubs/technical_reports/TR1176.html. 

This  RAND  report  evaluates  a  program  designed  to  help  veterans  reintegrate  back  into 
society  after  deployment  that  also  includes  their  families  and  loved  ones  in  the  process. 
The  program  aims  to  reduce  the  stigma  of  receiving  help  for  common  problems  service 
members  experience  upon  returning  home  from  deployment.  The  evaluation  rated  the 
media  and  other  content  of  the  program  on  how  well  they  reached  their  target  audience 
and  how  they  affected  veterans’  willingness  to  seek  help,  among  other  measures.  The 
review  of  the  methods  used  by  the  campaign,  as  well  as  its  goals  and  objectives,  might 
provide  an  interesting  comparison  for  IO  assessment. 

Altai  Consulting,  Afghan  Media  in  2010:  Synthesis  Report,  Washington,  D.C.: 

U.S.  Agency  for  International  Development,  October  13,  2010, 

http://www.altaiconsulting.com/docs/media/2010/Afghan%20Media%20in% 

202010.pdf. 

This  study,  prepared  for  the  U.S.  Agency  for  International  Development,  was  the  first 
comprehensive  media  evaluation  ever  conducted  in  Afghanistan.  The  study  endeavored 
to  create  a  comprehensive  picture  of  Afghan  media,  the  media  users,  and  their  prefer¬ 
ences  and  expectations  and  to  measure  media  impact  on  public  opinion  and  behavior. 
Research  conducted  from  March  to  August  2010  combined  quantitative  and  qualitative 
methodologies,  including  a  literature  review;  direct  observations;  key  informant  inter¬ 
views;  6,648  close-ended  interviews  in  107  districts,  covering  all  of  Afghanistan’s  34 
provinces;  a  daily  weeklong  audience  survey  of  more  than  1,500  individuals;  approxi¬ 
mately  200  qualitative,  open-ended  interviews;  and  ten  community  case  studies. 

British  Council,  Trust  Pays:  How  International  Cultural  Relationships  Build  Trust  in 
the  UK  and  Underpin  the  Success  of  the  UK  Economy,  2012, 
http://www.britishcouncil.org/organisation/publications/trust-pays. 

This  report  presents  findings  from  research  conducted  for  the  British  Council  (the 
UK’s  cultural  arm).  The  research  compared  the  attitudes  and  self-reported  behaviors 
of  those  who  had  contact  with  or  exposure  to  British  Council  products  and  activities 
with  a  control  group  of  individuals  who  did  not  have  that  exposure.  It  found  that  the 
amount  of  contact  study  participants  had  directly  affected  their  trust  in  the  UK. 
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Judy  VanSlyke  Turk  and  Linda  H.  Scanlan,  Evolution  of  Public  Relations:  Case 
Studies  of  Nations  in  Transition ,  3rd  ed.,  Gainesville,  Fla.:  Institute  for  Public 
Relations,  2008. 

This  case-study  volume  offers  examples  of  effective  and  ineffective  public  communica¬ 
tion  campaigns  in  developing  countries. 


Theories  of  Influence,  Academic 

As  noted  earlier,  one  of  the  foundations  of  evaluation  is  the  importance  of  a  theory 
of  change.  The  literature  on  psychology,  social  psychology,  and  the  study  of  influence 
includes  several  models  of  how  behaviors  and  attitudes  can  be  changed.  While  the  con¬ 
nection  to  assessment  is  only  tangential,  being  better  informed  about  possible  models 
for  understanding  influence  will  support  better  articulations  of  theories  of  change  for 
specific  programs  and  activities,  which  will  support  better  assessment. 

General  Overview 

William  D.  Crano  and  Radmilla  Prislin,  Attitudes  and  Attitude  Change,  New  York: 
Psychology  Press,  2008. 

This  book  provides  a  thorough  review  of  attitude  measurement,  including  explicit  and 
implicit  measurements,  attitude  change,  and  the  relationships  among  attitudes,  beliefs, 
and  behaviors.  It  reviews  past  issues  and  current  directions  of  theory  and  research  on 
attitudes  and  provides  brief  descriptions  regarding  multiple  theories  of  relevance  to 
attitudes  and  persuasion. 

Robert  H.  Gass  and  John  S.  Seiter,  Persuasion:  Social  Influence  and  Compliance 
Gaining,  5th  ed.,  New  York:  Pearson  Education,  2014. 

This  is  a  more  traditional  social  psychology  textbook  that  focuses  on  theories  and 
concepts  of  relevance  to  persuasion.  It  goes  into  depth  regarding  many  of  the  major 
theories  of  persuasion  and  addresses  factors  that  can  play  a  role  in  an  effort’s  persuasive 
impact.  It  is  perhaps  most  useful  as  a  source  for  theories  of  change  that  can  be  con¬ 
sidered  when  designing  inform,  influence,  and  persuade  efforts  and  their  assessments. 
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inputs  and  Outputs  in  Communication  Efforts 

William  J.  McGuire,  “McGuire’s  Classic  Input-Output  Framework  for 
Constructing  Persuasive  Messages,”  in  Ronald  Rice  and  Charles  Atkins,  eds.,  Public 
Communication  Campaigns ,  4th  ed.,  Thousand  Oaks,  Calif.:  Sage  Publications,  2012, 
pp.  133-146. 

This  chapter  describes  a  series  of  communication  factors  that  may  be  considered  in 
efforts  to  influence  an  audience.  It  also  proposes  a  rough  sequence  of  factors  that 
may  be  influenced  by  communication  factors.  This  matrix  suggests  variables  to  either 
manipulate  or  measure  when  designing  and  implementing  a  persuasive  effort. 

Elaboration  Likelihood  Model  and  the  Heuristic-Systematic  Model 

Alice  H.  Eagly  and  Shelly  Chaiken,  The  Psychology  of  Attitudes,  San  Diego,  Calif.: 
Harcourt  Brace  Jovanovich  College  Publishers,  1993. 

Eagly  and  Chaiken  present  the  “heuristic-systematic  model”  of  information  process¬ 
ing.  The  model  states  that  individuals  can  process  messages  in  one  of  two  ways:  heuris- 
tically  or  systematically.  The  implication  of  both  the  elaboration  likelihood  model  and 
the  heuristic-systematic  model  is  that  campaign  strategists  may  design  varied  campaign 
stimuli  that  operate  via  peripheral  and  central  routes  to  influence  attitude  change. 

Social  Learning  Theory 

Albert  Bandura,  Social  Foundations  of  Thought  and  Action:  A  Social  Cognitive  Theory, 
Upper  Saddle  River,  N.J.:  Prentice-Hall,  1986. 

According  to  Bandura’s  social  cognitive  learning  theory,  viewers  exposed  to  messages 
depicting  rewards  for  actions  exhibited  by  credible  role  models  are  more  likely  to  per¬ 
form  similar  behaviors. 

Processes  of  Opinion  Change 

Herbert  C.  Kelman,  “Processes  of  Opinion  Change,”  Public  Opinion  Quarterly, 

Vol.  25,  No.  1,  1961,  pp.  57-78. 

This  article  describes  a  classic  theory  in  social  psychology  regarding  the  processes  that 
lead  to  opinion  change.  They  include  compliance,  identification,  and  internalization. 
Each  process  involves  unique  methods  for  persuasion,  suggesting  that  the  desired  goal 
of  persuasion  should  guide  the  methods  used. 


Annotated  Inform,  Influence,  and  Persuade  Assessment  and  Evaluation  Reading  List  19 


Theory  of  Planned  Behavior 

leek  Ajzen  and  Martin  Fishbein,  Understanding  Attitudes  and  Predicting  Social 
Behavior,  Upper  Saddle  River,  N.J.:  Prentice-Hall,  1980. 

Ajzen  and  Fishbein’s  theory  of  reasoned  action  combines  attitudes  and  norms  with 
belief  expectancies  and  values  to  predict  behavioral  intentions.  Communication  cam¬ 
paigns  can  influence  beliefs  about  the  likelihood  of  experiencing  positive  and  negative 
outcomes  and  other  components  of  the  theory. 

Icek  Ajzen,  Dolores  Albarracin,  and  Robert  Hornik,  Prediction  and  Change  of  Health 
Behavior:  Applying  the  Reasoned  Action  Approach,  Mahway,  N.J.:  Lawrence  Erlbaum 
Associates,  2007. 

This  volume  extends  the  conceptual  parameters  of  the  theory  of  reasoned  action  into 
a  theory  of  planned  behavior,  providing  insight  into  the  pathways  of  influence  from 
campaign  messages  to  behavioral  outcomes. 

Cognitive  Dissonance 

Eddie  Harmon-Jones,  David  M.  Amodio,  and  Cindy  Harmon-Jones,  “Action-Based 
Model  of  Dissonance:  A  Review,  Integration,  and  Expansion  of  Conceptions  of 
Cognitive  Conflict,”  in  Mark  P.  Zanna,  ed.,  Advances  in  Experimental  Social 
Psychology,  Vol.  41,  Burlington,  Vt.:  Academic  Press,  2009,  pp.  119-166. 

This  chapter  provides  an  overview  of  the  theory  of  cognitive  dissonance,  one  of  the 
major  theories  of  attitude  change  in  social  psychology.  It  begins  with  a  description  of 
the  theory  and  then  moves  to  a  description  of  experiments  conducted  to  test  the  theory. 
It  then  describes  recent  advances  and  the  role  of  individual  and  cultural  differences  in 
cognitive  dissonance  and  related  theories.  Overall,  this  suggests  a  way  to  modify  audi¬ 
ence  attitudes  by  first  changing  behaviors. 

Knowledge,  Attitudes,  and  Practices 

Thomas  W.  Valente,  Patricia  Paredes,  and  Patricia  R.  Poppe,  “Matching  the 
Message  to  the  Process:  The  Relative  Ordering  of  Knowledge,  Attitudes,  and 
Practices  in  Behavior  Change  Research,”  Human  Communication  Research,  Vol.  24, 
No.  3,  March  1998,  pp.  366-385. 

This  article  provides  an  overview  of  the  different  processes  that  may  contribute  to  atti¬ 
tude  and  behavior  change.  It  notes  that  attitudes  do  not  always  precede  behaviors  and 
suggests  that  planners  should  consider  what  they  would  like  to  manipulate  and  how  it 
will  influence  a  particular  audience. 
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Principles  of  Social  Influence 

Robert  B.  Cialdini,  Influence:  Science  and  Practice,  Boston,  Mass.:  Allyn  and  Bacon, 

2001. 

In  this  fairly  accessible  examination  of  the  psychology  of  compliance,  the  author 
describes  factors  that  compel  an  individual  or  a  group  to  agree  with  the  requests  of 
others. 

Diffusion  of  Innovation  Theory 

Everett  M.  Rogers,  Diffusion  of  Innovations,  New  York:  Free  Press,  2010. 

Diffusion  theory  describes  how  new  ideas  and  practices  spread  through  interpersonal 
networks  and  the  influential  role  played  by  opinion  leaders.  Opinion  leaders  tend  to 
have  greater  exposure  to  media  messages  about  a  domain,  such  as  health,  and  are  more 
likely  to  exercise  informal  influence  over  the  attitudes  and  behaviors  of  friends  and 
family  members.  A  second  effects  strategy  is  to  initiate  an  indirect  or  multistep  flow  by 
disseminating  messages  to  potential  interpersonal  influencers  who  are  in  a  position  to 
personally  influence  individuals  of  interest. 

Other  Recent  Theories  of  Behavior  Change 

Joseph  N.  Cappella,  Martin  Fishbein,  Robert  Flornik,  R.  Kirkland  Ahern,  and 
Sarah  Sayeed,  “Using  Theory  to  Select  Messages  in  Anti-Drug  Media  Campaigns: 
Reasoned  Action  and  Media  Priming,”  in  Ronald  E.  Rice,  and  Charles  K.  Atkin, 
eds.,  Public  Communication  Campaigns,  3rd  ed.,  Thousand  Oaks,  Calif.:  Sage 
Publications,  2007,  pp.  323-342. 

The  integrative  theory  of  behavior  change  applies  a  combination  of  the  health  belief 
model,  social  cognitive  theory,  and  theory  of  reasoned  action  in  the  context  of  public 
communication  campaigns. 

B.  J.  Fogg,  “A  Behavior  Model  for  Persuasive  Design,”  Proceedings  of  the 
4th  International  Conference  on  Persuasive  Technology,  New  York:  ACM,  2009. 

This  paper  presents  the  Fogg  Behavioral  Model,  in  which  behavior  is  a  product  of 
three  factors  or  necessary  conditions  (motivation,  ability,  and  triggers),  each  of  which 
has  subcomponents.  For  a  person  to  perform  a  target  behavior,  the  three  factors  must 
occur  at  the  same  moment. 
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James  O.  Prochaska  and  Wayne  F.  Velicer,  “The  Transtheoretical  Model  of  Health 
Behavior  Change,”  American  Journal  of  Health  Promotion,  Vol.  12,  No.  1, 
September-October  1997,  pp.  38-48. 

The  transtheoretical  model  identifies  subgroups  based  on  their  current  stage  in  the  pro¬ 
cess  of  behavior  change.  The  stages  include  precontemplation,  contemplation,  prepara¬ 
tion,  action,  and  maintenance. 

Michael  D.  Slater,  “Integrating  Application  of  Media  Effects,  Persuasion,  and 
Behavior  Change  Theories  to  Communication  Campaigns:  A  Stages-of-Change 
Framework,”  Health  Communication,  Vol.  11,  No.  4,  1999,  pp.  335-354. 

The  “stages-of-change”  framework  integrates  the  application  of  media  effects,  persua¬ 
sion,  and  behavior  change  theories.  Over  time,  campaigns  target  specific  audience  seg¬ 
ments  at  different  stages  of  progression,  which  is  now  more  feasible  with  new  technolo¬ 
gies,  such  as  interactive  websites. 

Approaches  to  Organizing  and  Understanding  Efforts 

P.  Dolan,  M.  Hallsworth,  D.  Halpern,  D.  King,  R.  Metcalf,  and  I.  Vlaev, 

“Influencing  Behaviour:  The  MINDSPACE  Way,”  Journal  of  Economic  Psychology, 

Vol.  33,  No.  1,  February  2012,  pp.  264-277. 

MINDSPACE  is  a  mnemonic  developed  to  address  contextual  influences  of  behaviors. 
It  summarizes  the  different  elements  that  can  contribute  to  changes  in  automatic  pro¬ 
cesses.  It  also  provides  an  overview  of  subtle  methods  that  may  be  used  in  persuasion 
efforts. 

Susan  Michie,  Maartje  M.  van  Stralen,  and  Robert  West,  “The  Behaviour  Change 
Wheel:  A  New  Method  for  Characterizing  and  Designing  Behaviour  Change 
Interventions,”  Implementation  Science,  Vol.  6,  2011,  Article  42. 

The  behavior  change  wheel  provides  assistance  in  connecting  the  characteristics  of 
efforts  with  the  behaviors  that  the  efforts  seek  to  change.  The  wheel  can  assist  planners 
in  identifying  a  broad  approach  to  consider  as  part  of  their  persuasive  efforts. 


Theories  of  Influence,  Applied 

While  many  of  the  items  in  the  previous  section  veer  toward  the  dry,  abstract,  and 
academic,  the  items  in  this  section  are  much  more  practically  focused.  All,  however, 
remain  at  most  minimally  about  assessment  and  more  about  planning  and  executing 
influence. 


22  Assessing  and  Evaluating  DoD  Efforts  to  Inform,  Influence,  and  Persuade:  Annotated  Reading  List 


Andrew  Mackay  and  Steve  Tatham,  Behavioural  Conflict:  Why  Understanding  People 
and  Their  Motivations  Will  Prove  Decisive  in  Future  Conflict,  Essex,  UK:  Military 
Studies  Press,  2011. 

Written  by  British  psychological  operations  personnel,  this  book  argues  both  for  the 
importance  of  noncombatants  in  contemporary  and  future  conflict  environments  and 
for  prioritizing  behavioral  outcomes  over  attitudinal  outcomes  when  seeking  to  inform, 
influence,  and  persuade. 

Chip  Heath  and  Dan  Heath,  Made  to  Stick:  Why  Some  Ideas  Survive  and  Others  Die, 
New  York:  Random  House,  2007. 

This  book  features  a  highly  accessible  discussion  of  factors  that  make  ideas  and  mes¬ 
sages  “sticky”  and  durable. 

Jarol  B.  Manheim,  Strategy  in  Information  and  Influence  Campaigns:  How  Policy 
Advocates,  Social  Movements,  Insurgent  Groups,  Corporations,  Governments,  and  Others 
Get  What  They  Want,  New  York:  Routledge,  2011. 

This  is  a  more  strategic  or  operational-level  look  at  influence,  offering  an  introductory- 
level  discussion  of  the  various  theories  of  behavior  change  and  how  they  relate  to  an 
actual  influence  campaign. 

Alan  Kelly,  The  Elements  of  Influence:  Introducing  the  Playmaker’s  Standard:  The  New 
Essential  System  for  Managing  Competition,  Reputation,  Brand,  and  Buzz,  New  York: 
Dutton  Adult,  2006. 

Intended  for  a  business  context  (though  the  author  argues  that  the  framework  general¬ 
izes  to  defense),  this  volume  maps  out  a  series  of  possible  interactions  between  com¬ 
petitors  and  helps  identify  possible  responses,  called  “plays.”  It  is  an  interesting  way  of 
thinking,  if  nothing  else. 


Challenges  to  and  Criticisms  of  Assessment  Processes 

Turning  now  back  to  assessment,  the  following  resources  point  out  challenges  facing 
existing  assessment  practices.  All  provide  useful  cautions  to  keep  in  mind  when  devel¬ 
oping  assessments — cautions  that  should  also  be  kept  in  mind  when  designing  assess¬ 
ments  of  efforts  to  inform,  influence,  and  persuade. 
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Jonathan  Schroden,  “Why  Operations  Assessments  Fail:  It’s  Not  Just  the  Metrics,” 
Naval  War  College  Review,  Vol.  64,  No.  4,  August  2011,  pp.  89-102. 

According  to  Schroden,  the  problems  with  operations  assessments  run  much  deeper 
than  poor  metrics.  There  is  a  “failure  cycle”  at  work.  Key  challenges  that  need  to  be 
addressed  include  the  following: 

•  identifying  an  advocate  for  assessments 

•  fixing  planning  and  assessment  doctrine  so  that  it  provides  actual  guidance  on 
how  to  assess  and  not  just  a  vocabulary  or  definitions  (e.g.,  the  difference  between 
measures  of  performance  and  measures  of  effectiveness  is  interesting  but  not  help¬ 
ful,  operationally) 

•  creating  a  military  occupational  specialty  and  formal  course  of  instruction  for 
operations  assessment 

•  shifting  thinking  away  from  strictly  quantitative  and  picture-based  assessment 
products  toward  balanced,  comprehensive,  analytic  narratives. 

Stephen  Downes-Martin,  “Operations  Assessment  in  Afghanistan  Is  Broken,”  Naval 
War  College  Review,  Vol.  64,  No.  4,  Fall  2011,  pp.  103-124. 

Downes-Martin  rails  against  junk  arithmetic,  flawed  logic,  overoptimisim,  the  collec¬ 
tion  of  irrelevant  data,  and  many  other  problems  plaguing  contemporary  operations 
assessments.  He  offers  several  useful  pieces  of  advice  in  response. 

Tom  Watson  and  Paul  Noble,  Evaluating  Public  Relations:  A  Best  Practice  Guide  to 
Public  Relations  Planning,  Research,  and  Evaluation,  London:  Kogan  Page  Limited, 
2005. 

Watson  and  Noble  identify  the  following  barriers  to  evaluation: 

•  previous  working  experience  of  practitioners 

•  lack  of  knowledge  of  research  techniques 

•  the  manager-technician  dichotomy 

•  the  practitioners’  participation  in  decisionmaking 

•  lack  of  money  or  other  budgetary  issues. 

Ben  Connable,  Embracing  the  Fog  of  War:  Assessment  and  Metrics  in 
Counterinsurgency,  Santa  Monica,  Calif.:  RAND  Corporation,  MG-1086-DOD, 
2012,  http://www.rand.org/pubs/monographs/MG1086.html. 

This  monograph  lays  out  historical  and  contemporary  challenges  faced  by  efforts  to 
quantitatively  assess  complex  operations.  The  author  is  highly  critical  of  inappropriate 
quantification  and  inappropriate  aggregation  and  proposes  a  solution  based  on  narra¬ 
tive  assessments. 


The  U.S.  Department  of  Defense  has  struggled  to  assess  the  progress  and  effectiveness  of 
its  efforts  to  inform,  influence,  and  persuade  audiences  in  support  of  key  national  security 
objectives.  One  reason  is  that  it  lacks  personnel  with  sufficient  expertise  in  assessment  and 
evaluation.  Although  the  department  is  making  an  effort  to  infuse  sound  assessment  principles 
in  doctrine  and  to  expand  assessment-related  course  offerings  in  the  military-academic  sector, 
these  efforts  will  take  time  to  bear  fruit.  These  temporary  shortfalls  extend  to  the  evaluation 
and  assessment  of  DoD  efforts  to  inform,  influence,  and  persuade.  To  help  fill  the  gap,  RAND 
produced  a  reading  list  for  self-study  in  best  assessment  practices  across  a  range  of  sectors. 

The  reading  list  has  two  purposes:  to  provide  resources  for  new  assessment  personnel  to 
cement  and  broaden  their  assessment  and  evaluation  expertise  and  to  serve  as  a  general  list 
of  assessment  resources  that  can  be  made  available  to  assessment  stakeholders  to  improve 
their  assessment  expertise.  It  supplements  two  companion  volumes,  Assessing  and  Evaluating 
Department  of  Defense  Efforts  to  Inform,  Influence,  and  Persuade:  Desk  Reference  and  Assessing 
and  Evaluating  Department  of  Defense  Efforts  to  Inform,  Influence,  and  Persuade:  Handbook  for 
Practitioners. 


RAND 


NATIONAL  DEFENSE  RESEARCH  INSTITUTE 


www.rand.org 


$12.50 


ISBN-10  0-8330-8889-0 
ISBN-13  978-0-8330-8889-5 


780833  088895 


51250 


RR-809/3-OSD 


9 


